Visualization apparatus and program

ABSTRACT

The visualization apparatus comprises: a display that enables visualization of a predetermined information by superimposing the predetermined information in a three-dimensional space; a three-dimensional sensor that detects coordinate information of objects existing in the three-dimensional space within a display range of the display; a fitting unit that sequentially acquires coordinate information of each object detected by the three-dimensional sensor, matches the coordinate information of the object with a three-dimensional model of a target object prepared in advance for each acquisition to specify the target object from among the objects, and sequentially fits the three-dimensional model to the specified target object; and a display control unit that make the display to display a back surface shape of the target object detected by the three-dimensional sensor with respect to the front in a gaze direction by superimposing and displaying a three-dimensional model corresponding to a shape viewed from the front in the gaze direction on the target object.

CROSS-REFERENCE TO RELATED APPLICATION

This non-provisional application claims priority under 35 U.S.C. §119(a) from Japanese Patent Application No. 2019-086439, filed on Apr.26, 2019, and Japanese Patent Application No. 2020-040292, filed on Mar.9, 2020, the entire contents of which are incorporated herein byreference.

BACKGROUND Technical Field

The present invention relates to a visualization apparatus forvisualizing a back surface of a target object, and more particularly, toan apparatus effective for measurement by a three-dimensional measuringinstrument.

Background Art

As one of the methods for measuring a three-dimensional shape of anarticle, there is a method using a three-dimensional measuringinstrument (e.g., see JP-A-2000-65561). An exemplary three-dimensionalmeasuring instrument is shown in FIG. 1A. The three-dimensionalmeasuring instrument 10 is roughly divided into a main body 10 a and aprobe 10 b having a stylus at tip thereof. The target object W of themeasurement is placed on the main body 10 a, and the probe 10 b isattached to the main body 10 a so as to be movable in thethree-dimensional direction by a drive mechanism provided in the mainbody 10 a.

The three-dimensional measuring instrument 10 measures thethree-dimensional coordinate value of the contact position by bringingthe tip of the probe into contact with the surface of the target objectW. In addition to the contact type, there is also a non-contact typeprobe such as an optical type which measures the surface position byphotographing the surface of the target object W from an arbitrarythree-dimensional space position.

SUMMARY OF THE INVENTION Problems to be Solved by the Invention

For example, as shown in FIG. 1B, when the three-dimensional measuringinstrument 10 measures the depression D on the front face of the targetobject W as viewed from an operator, the depression D can be visuallyrecognized from the viewpoint of the operator. In addition, the tip 10bt of the probe 10 b in front of the target object W as viewed from theoperator can also be visually recognized. Therefore, the tip 10 bt canbe brought into appropriate contact with the depression D. However, asshown in FIG. 1C, when the shape of the back surface of the targetobject W is measured without changing the arrangement of the targetobject W, the shape of the back surface cannot be visually recognizedfrom the viewpoint of the operator. Further, since it is necessary toarrange the probe 10 b behind the target object W, the tip 10 bt cannotbe visually recognized. Therefore, when measuring the back surface, thearrangement of the target object W is generally changed (for example,inverted) so that the back surface can be measured.

However, when the arrangement of the target object is changed, thecoordinate system must be reset so that the measurement coordinates onthe object after the position change match the measurement coordinatesat the same position before the position change. This causes problemssuch as a long measurement time.

It is an object of the present invention to provide a visualizationapparatus and program that enable visualization of a back surface shapeof a target object with respect to the front in a gaze direction of theoperator as a shape viewed from the front in a gaze direction.

Means for Solving the Problems

The visualization apparatus of the present invention comprises: adisplay that displays predetermined information superimposed on athree-dimensional space to make the predetermined information visible; athree-dimensional sensor that detects coordinate information of objectsexisting in the three-dimensional space within a field of view throughthe display; a fitting unit that sequentially acquires coordinateinformation of each object detected by the three-dimensional sensor,matches the coordinate information of the object with athree-dimensional model of a target object prepared in advance for eachacquisition to specify the target object from among the objects, andsequentially fits the three-dimensional model to the specified targetobject; and a display control unit that make the display to display asee-through region in the three-dimensional model so as to besuperimposed on a predetermined gaze area of the target object. Thesee-through region is corresponding to the predetermined gaze area ofthe surface of the target object existing within the field of viewthrough the display. The see-through region is seen through from thesurface area of the three-dimensional model superimposed on the targetobject in a gaze direction from the three-dimensional sensor toward theapproximate center of the predetermined gaze area. The display controlunit may make the display to display the see-through region superimposedon a predetermined gaze area of the target object by transparentizingthe region except the see-through region in the three-dimensional modelsuperimposed on the target object. The predetermined gaze area may be,for example, a region of the surface of the target object to be visuallyrecognized in a predetermined region in the display. The predeterminedgaze area may be a predetermined region on the surface of the targetobject, centered on the intersection of a line from the predeterminedposition on the display toward the gaze direction and the surface of thetarget object. Here, the “predetermined position of the display” may be,for example, the center of the display. The “gaze direction” may be, forexample, a direction perpendicular to the screen of the display, adirection of the line of sight, a direction at which the tip of theprobe exists, or the like.

As a result, by viewing the surface of the target object through thedisplay, the shape of the back surface of the target object can be seenthrough in a simulated manner. The display may be a transmissive displaythat allows the three-dimensional space to be viewed through the displayscreen, or a non-transmissive display that allows the three-dimensionalspace to be viewed by displaying an image of the three-dimensional spacecaptured by a camera.

The visualization apparatus may further comprise: a measuring instrumentinformation acquisition unit for acquiring information on thethree-dimensional model and tip coordinates of a probe of apredetermined measuring instrument in which the measuring coordinatesystem is set to be the same as the detecting coordinate system of thethree-dimensional sensor by a predetermined method; and a probe portionspecifying unit for specifying, when the tip coordinates of the probeare included in a coordinate range behind the target object that sweepsthrough the predetermined gaze area of the target object in the gazedirection, the portion of the probe included in the coordinate range andthe existence position of the portion based on the tip coordinates andthe three-dimensional model of the probe. Then, the display control unitmay superimpose a portion of the three-dimensional model of the probe ona predetermined gaze area of the target object so that a portion of thethree-dimensional model of the probe corresponding to the portion of theprobe is visually recognized at a position behind the see-through regioncorresponding to the existence position of the portion of the probespecified by the probe portion specifying unit. In this case, thepredetermined gaze area may be, for example, a predetermined area of thesurface of the target object centered on an intersection point between aline connecting the predetermined position of the display and the tip ofthe probe and the surface of the target object.

As a result, the measurement of the back surface of the target object bythe measuring instrument can be performed from the front surface sidewhile the position of the probe behind the target object and the backsurface shape of the target object are seen through in a simulatedmanner.

A visualization apparatus according to another embodiment of the presentinvention comprises: a display that enables visualization of apredetermined information by superimposing the predetermined informationin a three-dimensional space; a three-dimensional sensor that detectscoordinate information of objects existing in the three-dimensionalspace within a display range of the display; a fitting unit thatsequentially acquires coordinate information of each object detected bythe three-dimensional sensor, matches the coordinate information of theobject with a three-dimensional model of a target object prepared inadvance for each acquisition to specify the target object from among theobjects, and sequentially fits the three-dimensional model to thespecified target object; and a display control unit that make thedisplay to display a back surface shape of the target object detected bythe three-dimensional sensor with respect to the front in a gazedirection by superimposing and displaying a three-dimensional modelcorresponding to a shape viewed from the front in the gaze direction onthe target object.

Thus, by viewing the front of the target object through the display, thebuck surface shape of the target object can be seen through in asimulated manner. The display may be a transmissive display that allowsthe three-dimensional space to be viewed through the display screen, ora non-transmissive display that allows the three-dimensional space to beviewed by displaying an image of the three-dimensional space captured bya camera.

In the present invention, the visualization apparatus may furthercomprises: a coordinate measuring instrument information acquisitionunit for acquiring information of the tip coordinate of a probe in apredetermined measuring instrument in which the measuring coordinatesystem is set to be the same as the detection coordinate system of thethree-dimensional sensor according to a predetermined method; and aprobe portion specifying unit for specifying a portion of the probeoverlapping the three-dimensional model of the back surface shape whenthe probe is present behind the three-dimensional model superimposed onthe target object as the back surface shape of the target object viewedfrom the front in the gaze direction. The fitting unit may fit thethree-dimensional model of the probe to the portion of the probespecified by the probe portion specifying unit. Further, the displaycontrol unit may make the three-dimensional model corresponding to theback surface shape of the target object viewed from the front in thegaze direction, and the three-dimensional model fitted to the portion ofthe probe present therebehind be visibly distinguished.

In the present invention, when the three-dimensional model correspondingto the shape seen from the front in the gaze direction is superimposedon the target object, the display control unit may not display thesuperimposed display of the region other than the specific range in thegaze direction.

Here, as a specific example of “visibly distinguished”, the back surfaceshape may be displayed in the form of a grayscale display color or awire frame format, while the portion of the probe may be displayed in atrue color display color or in a texture map format so as to give atexture to the surface of the three-dimensional model. Thereby the backsurface shape and the portion of the probe may be superimposed so thatthey can be clearly distinguished.

The functions of the respective units of the visualization apparatus ofthe present invention may be realized by being described in a programand executed by a computer.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A, 1B and 1C are diagrams illustrating problems in theconventional three-dimensional measurement.

FIG. 2. is a functional block diagram of the visualization apparatus 100of the present invention.

FIGS. 3A to 3D are diagrams illustrating a process of specifying atarget object in a field of view of the transmissive display, fittingthe three-dimensional model, and specifying a gaze area.

FIG. 4 is a diagram illustrating a see-through region.

FIGS. 5A to 5D are diagrams illustrating a principle of making the backsurface shape of the object being able to be seen through in a simulatedmanner.

FIG. 6 is a diagram showing an example of a back shape of the objectbeing seen through in a simulated manner.

FIG. 7 is a functional block diagram of the visualization apparatus 200of the present invention.

FIG. 8 is a diagram illustrating a principle of making the back surfaceshape of the target object and a portion of the probe behind the targetobject being able to be seen through in a simulated manner.

FIG. 9 is a diagram showing an example of the back surface shape of thetarget object and the portion of the probe behind the target objectbeing seen through in a simulated manner.

FIG. 10 is a diagram illustrating an exemplary configuration of thevisualization apparatus 100 or the visualization apparatus 200 of thepresent invention that realizes the functions of respective units byexecuting a program in which the functions of respective units aredescribed by a CPU.

FIGS. 11A to 11C are diagrams showing an example in which thevisualization apparatus of the present invention is applied to athree-dimensional measuring instrument employing an articulated armsystem and a non-contact type probe.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Embodiments of the present invention will be described below withreference to the drawings. In the following description, the same partsare denoted by the same reference numerals, and the description of theparts once described is omitted as appropriate.

First Embodiment

FIG. 2 is a functional block diagram of the visualization apparatus 100of the present invention. The visualization apparatus 100 includes atransmissive display 110, a three-dimensional sensor 120, a fitting unit130, and a display control unit 140.

The transmissive display 110 is a display device of any type in whichpredetermined information is superimposed and displayed in athree-dimensional space behind itself so that the wearer can visuallyrecognize the information.

The three-dimensional sensor 120 is a detection device of any type thatdetects coordinate information of objects existing in a field of view ofa three-dimensional space through the transmissive display 110. Thecoordinate information of each object is detected as three-dimensionalcoordinates of each of a plurality of feature points characterizing theshape of the object.

The fitting unit 130 sequentially acquires coordinate information ofeach object detected by the three-dimensional sensor 120. Then, eachtime the coordinate information is acquired, the target object W isspecified from among the objects by matching with the three-dimensionalmodel of the target object W stored in advance in, for example, thestorage unit 101 or the like, and the three-dimensional model issequentially fitted to the specified target object W. Thethree-dimensional model of the target object W comprises a plurality offeature point information characterizing the shape of the target objectW.

In the three-dimensional sensor 120, feature point information can beobtained only on a surface portion which can be detected at an existenceposition of the target object W. However, by applying thethree-dimensional model of the target object W to the existence positionof the target object W using the obtained feature point information as aclue, it is possible to supplement the feature point information of theportion which cannot be detected by the three-dimensional sensor 120.Therefore, it is possible to obtain the feature point information of theentire target object W at the existence position of the target object W.In the present invention, this process is referred to as fitting. Due tothe nature of such fitting, the fitting of the three-dimensional modelto the target object W is not necessarily performed visually, and thetarget object W may be kept in a state in which target the object W isseen as it appears as long as the coordinates of the portion at whichthe coordinates are not detected by the three-dimensional sensor 120 arerelatively specified in relation to the portion at which the coordinatesare specified.

The display control unit 140 makes the display to display a back surfaceshape of the target object detected by the three-dimensional sensor withrespect to the front in a gaze direction by superimposing and displayinga three-dimensional model corresponding to a shape viewed from the frontin the gaze direction on the target object. In addition, the displaycontrol unit 140 makes it possible to hide superimposed display ofregions other than a specific range in the gaze direction. As a specificprocess, the display control unit 140 may cause the transmissive display110 to display a see-through region in the three-dimensional model bysuperimposing the see-through region on a predetermined gaze area of thetarget object W. Here, the see-through region in the three-dimensionalmodel refers to a region corresponding to the predetermined gaze area ofthe surface of the target object W existing within the field of viewthrough the transmissive display 110. The region is seen through fromthe surface area of the three-dimensional model superimposed on thetarget object in a gaze direction from the three-dimensional sensortoward the approximate center of the predetermined gaze area. Thepredetermined gaze area is, for example, a region of the surface of thetarget object W visually to be recognized in a predetermined region inthe transmissive display 110.

Specific examples will be described referring to FIGS. 3A to 3D and FIG.4.

As shown in the FIG. 3A, when part or all of the objects U is capturedin the field of view FV through the transmissive display 110, thethree-dimensional sensor 120 acquires coordinate information indicatingthe shape of the objects U, and collates the coordinate information withthe three-dimensional model WM of the target object W.

When a certain object U is identified as the target object W by thecollation (FIG. 3B), the fitting unit 130 fits the three-dimensionalmodel WM to the target object W as shown in FIG. 3C. In FIG. 3C, thethree-dimensional model WM is visually fitted on the target object W ina superimposed manner, but the fitting may not be visibly performed asdescribed above.

Subsequently, by changing the orientation of the transmissive display110 or the like, the surface of the target object W is captured in anactivation area Ta set in the center portion of the transmissive display110 as shown in FIG. 3D. Hereinafter, the surface range of the targetobject W captured in the activation area Ta is referred to as a gazearea Ga, and a direction from the existing position PV of thethree-dimensional sensor 120 toward a predetermined position within thegaze area Ga, for example, the substantially center Gc is referred to asa gaze direction VD.

The fitting of the three-dimensional model WM to the target object W isnot necessarily performed at the timing when the object U is specifiedas the target object W, and may be performed, for example, at the timingwhen the surface of the target object W is captured in the activationarea Ta of the transmissive display 110.

The information of the coordinate range of the field of view FV by thetransmissive display 110 is successively detected by thethree-dimensional sensor 120, and the position of the activation area Tain the field of view FV is also specified in advance. For this reason,it is possible to specify a coordinate range for a portion within thefield of view FV captured by the activation area Ta, for example, thegaze area Ga on the object W. Further, since the existence position PVof the three-dimensional sensor 120 is known in the three-dimensionalsensor 120, and the approximate center Gc of the gaze area Ga is alsospecified within the coordinate range of the gaze area Ga, the gazedirection VD can also be specified.

The display control unit 140 causes the transmissive display 110 todisplay the see-through region FD, which is seen through in the gazedirection VD from the surface area Gam corresponding to the gaze areaGa, of the three-dimensional model WM fitted to the target object W, bysuperimposing the see-through region FD on the object W.

As shown in FIG. 4, the see-through region FD is defined as a columnarregion that through which the surface area Gam of the three-dimensionalmodel WM corresponding to the gaze area Ga passes when sweeping in thethree-dimensional model WM in the gaze direction VD. For example, if thesurface area Gam is circular, the see-through region FD has acylindrical shape.

As described above, the coordinate range of the gaze area Ga (surfacearea Gam) and the gaze direction VD are specified, and the existencecoordinate range of the target object W (and three-dimensional model WM)is also specified by the detection of the feature points by thethree-dimensional sensor 120 and the fitting of the three-dimensionalmodel WM. Therefore, the coordinate range of the see-through region FDcan be specified based on these pieces of information.

A specific method of superimposing the see-through region FD on thetarget object W and displaying it on the transmissive display 110 isarbitrary. For example, with respect to the three-dimensional model WMsuperimposed and displayed on the target object W by fitting, only thesee-through region FD from the surface area Gam may be superimposed anddisplayed on the gaze area Ga of the target object W by transparentizinga region other than the see-through region FD from the surface area Gam.In addition, only the see-through region FD from the surface area Gammay be superimposed on the gaze area Ga of the target object W withrespect to the three-dimensional model WM which is fitted to the targetobject W but is not displayed.

Owing to the visualization apparatus 100 of the present inventionconfigured as described above, by viewing the target object W from thefront face through the transmissive display 110, it is possible to seethrough the back surface shape of the target object W in a simulatedmanner according to the following principle.

FIG. 5A is a diagram showing examples of the back surface Wb of thetarget object W when the target object W shown in FIG. 4 is seen fromthe back surface. The back surface area Gb shown in the drawing 5A is aregion which comes out of the back surface Wb when the gaze area Ga isswept in the gaze direction VD, and is the other bottom surface which ispaired with the surface area Gam (gaze area Ga) which corresponds to onebottom surface of the see-through region FD which is a columnar region.

Two depressions D1 and D2 exist in the back surface area Gb. Therefore,depressions corresponding to the depressions D1 and D2 also exist on theback surface of the three-dimensional model WM of the target object W.

When the target object W is a three-dimensional object filled withcontents, the depressions D1 and D2 existing in the back surface area Gband the back surface area Gb cannot be visually observed from the frontsurface side. Therefore, assuming that can be observed from the surfaceside, the projections P1 and P2 corresponding to the depressions D1 andD2 are observed at a position shown in FIG. 5B in a virtual area Gbbcorresponding to the back surface area Gb on the virtual surface Wbbcorresponding to the back surface Wb.

When the three-dimensional model WM is displayed in a wireframerepresentation at the existence position of the target object W detectedin the field of view FV through the transmissive display 110, theprojections P1 and P2 on the virtual surface Wbb as shown in FIG. 5C canbe observed. However, when the wearer of the transmissive display 110performs some work, it is generally assumed that the wearer uses asubstantially central portion of the field of view FV, and that gazes ata portion of the target object W captured in the central portion.

Therefore, focusing on this, the three-dimensional models WM arevisualized and superimposed on the target object W only in thesee-through region FD from the gaze area Ga to the gaze direction VDspecified on the target object W as shown in FIG. 5D. As a result, whenthe wearer of the transmissive display 110 directs his/her gaze to thetarget object W, as shown in FIG. 6, the wearer can observe the virtualsurface Wbb inside the target object W as if he/she were seeing throughfrom the gaze area Ga in the gaze direction VD.

Second Embodiment

FIG. 7 shows a functional block diagram of the visualization apparatus200 of the present invention. The visualization apparatus 200 issuitable for measurement performed in a state where the target object Wis placed on the main body 10 a of the three-dimensional measuringinstrument 10 including the main body 10 a, the probe 10 b, and thecontrol unit 10 c. It is assumed that the measurement coordinate systemof the three-dimensional measuring instrument 10 is set to be the sameas the detection coordinate system of the three-dimensional sensor 120by an arbitrary method. Although the case where the measuring instrumentis a three-dimensional measuring instrument will be described as anexample, the present invention can be applied to other measuringinstruments using a probe connected by wire or wireless in the samemanner.

When a part of the probe 10 b (particularly the tip) is hidden behindthe target object W placed on the three-dimensional measurementapparatus 10 and cannot be seen, the visualization apparatus 200visualizes at least the tip of the probe 10 b in a simulated manner andvisualizes the back surface shape of the target object W in a simulatedmanner. Among these, the visualization in the simulated manner of theback surface shape of the target object W is realized by a functionalunit corresponding to the visualization apparatus 100 of the firstembodiment included in the visualization apparatus 200.

The visualization apparatus 200 includes a transmissive display 110, athree-dimensional sensor 120, a fitting unit 130, a display control unit240, a measuring instrument information acquisition unit 250, and aprobe portion specifying unit 260.

The measuring instrument information acquisition unit 250 acquires thethree-dimensional model and coordinate information of the probe 10 b ofthe three-dimensional measuring instrument 10. The three-dimensionalmodel is acquired from an arbitrary storage means in which thethree-dimensional model is stored in advance, for example, the storageunit 101. The coordinate information is acquired from thethree-dimensional measuring instrument 10. The method of acquiring thethree-dimensional model and the coordinate information from thearbitrary storage means and the three-dimensional measuring instrument10 are arbitrary. When the information is acquired from thethree-dimensional measuring instrument 10, the information can beacquired by wireless communication or wired communication of any typevia the control unit 10 c, for example.

The probe portion specifying unit 260 specifies a portion of the probewhere the three-dimensional model having the back surface shape overlapswhen the probe exists behind the three-dimensional model superimposedand displayed as the back surface shape of the target object W viewedfrom the front in gaze direction. As a specific process, when thecoordinates of the tip of the probe 10 b are included in the coordinaterange behind the target object W passing by sweeping the gaze area ofthe target object W in the gaze direction, the probe portion specifyingunit 260 may specify the portion of the probe 10 b included in thecoordinate range behind the target object W and the existence positionof the portion based on the coordinates of the tip and thethree-dimensional model of the probe 10 b.

The display control unit 240 has the following functions in addition tothe functions of the display control unit 140. The display control unit240 superimposes and displays a portion of the three-dimensional modelof the probe 10 b on a predetermined gaze area of the target object W sothat a portion of the three-dimensional model corresponding to theportion of the probe 10 b is visually recognized at a position behindthe see-through region corresponding to the position where the portionof the probe 10 b specified by the probe portion specifying unit 260exists. For example, the back surface shape may be displayed in agrayscale display color or a wire frame format, and the portion of theprobe may be displayed in a true color display color or in a texturemapping format so as to give a texture to the surface of thethree-dimensional model. Thereby the back surface shape and the portionof the probe may be superimposed so that they can be clearlydistinguished.

A specific example will be described with reference to FIG. 8.

The probe portion specifying unit 260 determines whether or not thecoordinates of the tip 10 bt of the probe 10 b are included in thecoordinate range BD behind the target object W passing by sweeping thegaze area Ga in the gaze direction VD (or in the coordinate range BDbehind the see-through region FD). Then, when it is determined that thecoordinates are included in the coordinate range BD, The probe portionspecifying unit 260 specifies the portion 10 bp of the probe 10 bincluded in the coordinate range BD behind the target object W and theexistence position of the portion 10 bp based on the coordinates of thetip 10 bt and the three-dimensional model PM of the probe 10 b.

Here, the coordinate range BD behind the target object W can bespecified based on the gaze area Ga, the coordinate range of the targetobject W, and the gaze direction VD that have already been specified inthe transparentizing process of the target object W. In addition, sincethe orientation and the movable range of the probe 10 b in themeasurement coordinate system of the three-dimensional measuringinstrument 10 are specified, the existing coordinate range of the probe10 b can be specified based on the coordinates of the tip 10 bt of theprobe 10 b and the three-dimensional model of the probe 10 b acquired bythe measuring instrument information acquisition unit 250. Therefore,the portion 10 bp of the probe 10 b and the existence position of theportion 10 bp can be specified as a portion where the existencecoordinate range of the probe 10 b overlaps with the coordinate range BDbehind the target object W and the existence position of the portion.

Then, the display control unit 240 superimposes and displays the portionof the three-dimensional model PM of the probe 10 b on the gaze area Gaof the target object W so that the portion of the three-dimensionalmodel PM of the probe 10 b corresponding to the portion 10 bp of theprobe 10 b is visually recognized at the position corresponding to theexistence position of the portion bp of the probe 10 b behind the viewarea FD.

According to the visualization apparatus 200 of the present inventiondescribed above, when the wearer of the transmissive display 110 directshis/her gaze to the target object W, as shown in FIG. 9, it is possibleto observe the virtual surface Wbb inside the target object W and theportion 10 bp of the probe 10 b behind the virtual surface Wbb as ifthey were being seen through in the gaze direction VD from the gaze areaGa. Therefore, the measurement of the back surface of the target objectW by the three-dimensional measuring instrument 10 can be performed fromthe front surface side while checking the position of the probe 10 b andthe back surface shape of the target object W.

In the visualization apparatus 200 of the second embodiment, the gazearea Ga may be a predetermined range of the surface of the target objectW centered on the intersection of the line connecting a predeterminedposition of the transmissive display (for example the center of thedisplay) and the tip 10 bt of the probe 10 b and the surface of thetarget object W, for example. As a result, the gaze area Ga can bechanged in accordance with the movement of the probe 10 b or the wearerof the transmissive display 110.

Third Embodiment

The functions of each unit of the visualization apparatus 100 or thevisualization apparatus 200 may be realized by being described in aprogram and executed by a computer.

FIG. 10 shows an exemplary configuration of the visualization apparatus100 or the visualization apparatus 200 when the functions of therespective means are written in a program and executed by a computer.

The visualization apparatus 100 or the visualization apparatus 200includes, for example, a storage unit 101, a CPU102, a communicationunit 103, a transmissive display 110, and a three-dimensional sensor120.

The CPU102 executes programs stored in the storage unit 101 and realizesthe functions of the visualization apparatus 100 or the visualizationapparatus 200. The functions of the respective units are described inthe programs. The storage unit 101 is an arbitrary storage unit thatstores a three-dimensional model or a program, and can employ, forexample, a nonvolatile memory, a volatile memory, or the like inaddition to a storage medium such as an HDD or a flash memory. Insteadof being provided in the visualization apparatus 100 or thevisualization apparatus 200, the storage unit 101 may be realized byemploying a cloud storage connected via the communication unit 103. Thecommunication unit 103 is an interface for connecting to a wirelessnetwork or a wired network, and transmits/receives information to/fromthe control unit 10 c of the three-dimensional measuring instrument 10connected to the network, a cloud storage, or the like in accordancewith control by programs executed by the CPU102. The transmissivedisplay 110 and the three-dimensional sensor 120 display information anddetect an object respectively, under control of programs executed byCPU102.

The present invention is not limited to the above embodiments. Eachembodiment is exemplified, and any embodiment having substantially thesame constitution as the technical idea described in the claims of thepresent invention and exhibiting the same operation and effect isincluded in the technical scope of the present invention. That is, thepresent invention can be suitably modified within the scope of thetechnical idea expressed in the present invention, and forms to whichsuch modifications and improvements are added are also included in thetechnical scope of the present invention.

For example, in each of the embodiments described above, the shape ofthe three-dimensional measuring instrument and the method of the probehave been described by taking an example in which the shape is a gatetype and the probe is a contact type, but the present invention can beapplied to other shapes and other types.

For example, even in the case of the three-dimensional measuringinstrument 10 of the multi-joint arm type and the probe of thenon-contact type as shown in FIG. 11A, a configuration in which whenmeasuring the surface of the target object W, the measurement isperformed while visual recognized the surface as shown in FIG. 11B, andwhen measuring the back surface, the vicinity of the tip 10 bt of theprobe 10 b is seen-through in a simulated manner as shown in FIG. 11Ccan be realized in substantially the same manner as in each of the aboveembodiments.

In each of the above-described embodiments, a case has been described inwhich a transmissive display capable of visually recognizing athree-dimensional space by transmitting through a display screen isemployed as an example of the display, but the display may be anon-transmissive display. In this case, it may be possible to visuallyrecognize the three-dimensional space by displaying an image of thethree-dimensional space taken by the camera. As a specific configurationexample employing a non-transmissive display, it may be superimposed anddisplayed predetermined information in a three-dimensional space in adevice capable of augmented reality (AR) display such as a smartphone ora virtual reality (VR) device that enables so-called video see-throughby combining a head-mounted display and a camera.

The method of determining the gaze area is not limited to the methodsshown in the above embodiments. For example, a predetermined gaze areamay be a predetermined region of surface of the target object, centeredon the intersection of a line from the predetermined position on thedisplay toward the gaze direction and the surface of the target object.Here, the “predetermined position on the display” may be, for example,the center of the display. The “gaze direction” may be, for example, adirection perpendicular to the screen of the display, a direction of theline of sight, a direction at which the tip of the probe exists, and thelike. When the gaze direction is the direction of the gaze, thevisualization device may be provided with a sensor for detecting thedirection of the line of sight of the wearer, and the direction of theline of sight detected by the sensor may be the “gaze direction”.

What is claimed is:
 1. A visualization apparatus comprising: a displaythat displays predetermined information superimposed on athree-dimensional space to make the predetermined information visible; ameasuring instrument that measures a three-dimensional shape of anobject; a three-dimensional sensor that detects coordinate informationof the object existing in the three-dimensional space within a field ofview through the display; a fitting unit for sequentially acquirescoordinate information of each object detected by the three-dimensionalsensor, matches the coordinate information of the object with athree-dimensional model of a target object prepared in advance for eachacquisition to specify the target object from among the objects, andsequentially fits the three-dimensional model to the specified targetobject; a display control unit that make the display to display asee-through region in the three-dimensional model so as to besuperimposed on a predetermined gaze area of the target object, thesee-through region corresponding to the predetermined gaze area of thesurface of the target object existing within the field of view throughthe display, and the see-through region being seen through from thesurface area of the three-dimensional model superimposed on the targetobject in a gaze direction from the three-dimensional sensor toward theapproximate center of the predetermined gaze area; a measuringinstrument information acquisition unit for acquiring information on thethree-dimensional model and tip coordinates of a probe of thepredetermined measuring instrument in which the measuring coordinatesystem is set to be the same as the detecting coordinate system of thethree-dimensional sensor by a predetermined method; and a probe portionspecifying unit for specifying, when the tip coordinates of the probeare included in a coordinate range behind the target object that sweepsthrough the predetermined gaze area of the target object in the gazedirection, the portion of the probe included in the coordinate range andthe existence position of the portion based on the tip coordinates andthe three-dimensional model of the probe, wherein the display controlunit superimposes and displays a portion of the three-dimensional modelof the probe on a predetermined gaze area of the target object so that aportion of the three-dimensional model of the probe corresponding to theportion of the probe is visually recognized at a position behind thesee-through region corresponding to the existence position of theportion of the probe specified by the probe portion specifying unit. 2.The visualization apparatus according to claim 1, wherein the displaycontrol unit makes the display to display the see-through regionsuperimposed on a predetermined gaze area of the target object bytransparentizing the region except the see-through region in thethree-dimensional model superimposed on the target object.
 3. Thevisualization apparatus according to claim 1, wherein the predeterminedgaze area is a region of a surface of the target object that is viewedin a predetermined region in the display.
 4. The visualization apparatusaccording to claim 3, wherein the predetermined gaze area is apredetermined are of the surface of the target object centered on theintersection of a line from a predetermined position on the displaytoward the gaze direction and the surface of the target object.
 5. Thevisualization apparatus according to claim 1, wherein the predeterminedgaze area is a predetermined region of the surface of the target objectcentered on an intersection point between a line connecting apredetermined position on the display and the tip of the probe and thesurface of the target object.
 6. A non-transitory computer-readablestorage medium having stored thereon executable instructions that whenexecuted by processor of computer control the computer to perform aseach unit constituting the visualization apparatus according to claim 1.7. A visualization apparatus comprising: a display that enablesvisualization of a predetermined information by superimposing thepredetermined information in a three-dimensional space; a measuringinstrument that measures a three-dimensional shape of objects; athree-dimensional sensor that detects coordinate information of objectsexisting in the three-dimensional space within a display range of thedisplay; a fitting unit that sequentially acquires coordinateinformation of each object detected by the three-dimensional sensor,matches the coordinate information of the object with athree-dimensional model of a target object prepared in advance for eachacquisition to specify the target object from among the objects, andsequentially fits the three-dimensional model to the specified targetobject; a display control unit that make the display to display a backsurface shape of the target object detected by the three-dimensionalsensor with respect to the front in a gaze direction by superimposingand displaying a three-dimensional model corresponding to a shape viewedfrom the front in the gaze direction on the target object; a coordinatemeasuring instrument information acquisition unit for acquiringinformation of the tip coordinate of a probe in a predeterminedmeasuring instrument in which the measuring coordinate system is set tobe the same as the detection coordinate system of the three-dimensionalsensor according to a predetermined method; and a probe portionspecifying unit for specifying a portion of the probe overlapping thethree-dimensional model of the back surface shape when the probe ispresent behind the three-dimensional model superimposed on the targetobject as the back surface shape of the target object viewed from thefront in the gaze direction, wherein the fitting unit fits thethree-dimensional model of the probe to the portion of the probespecified by the probe portion specifying unit, and the display controlunit makes the three-dimensional model corresponding to the back surfaceshape of the target object viewed from the front in the gaze directionand the three-dimensional model fitted to the portion of the probepresent therebehind be visibly distinguished.
 8. The visualizationapparatus according to claim 7 wherein, when the three-dimensional modelcorresponding to the shape seen from the front in the gaze direction issuperimposed on the target object, the display control unit hides thesuperimposed display of the region other than the specific range in thegaze direction.
 9. A non-transitory computer-readable storage mediumhaving stored thereon executable instructions that when executed byprocessor of computer control the computer to perform as each unitconstituting the visualization apparatus according to claim 7.